New Directions in Adaptive Disk Reorganization

While processor clock rates continue to track Moore’s law, the physical mechanisms which contribute to disk access time are improving at a much slower pace. As a result, disk access time is getting longer relative to CPU cycles. Caching can alleviate congestion by keeping frequently used data in main memory, but these data must initially read from disk at some point. Furthermore, the entire working set will not always fit in main memory. In such a scenario, not only can the file buffer cache fail to be effective, but the contents of main memory themselves may be swapped to disk. Accordingly, it pays to reduce the cost of disk operations. One technique to mitigate the problem is efficient data arrangement: strategically placing data on the disk in order to minimize expected access time. Since this strategy relies on knowledge about access patterns, the arrangement cannot generally be determined when the data are first written. Instead, a common approach is to gather statistics and reorganize the data to reduce expected access time given the observed pattern [3, 16, 11, 13, 2]. Statistics gathering and data relocation are not free, however. It is important to view data arrangement in the proper context of other mechanisms. File caching is the most important of these; it has smaller overhead cost and a much larger potential payoff than data reorganization, because a cache hit bypasses disk access entirely. File caches cannot handle all requests, however, so a good reorganization scheme should work in conjunction with file caching to reduce access latency. Other considerations for the design of a reorganization system are the way in which it affects the effectiveness of the disk prefetch cache and the way it interacts with disk scheduling algorithms.